DOCUMENT RESUME 



ED 358 135 



TM 019 919 



AUTHOR 
TITLE 

PUB DATE 
NOTE 

PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Marsh, Herbert W. ; Grayson, David 
The Composite Direct Product Model: Strengths, 
Limitations, and New Approaches. 
2 Apr 93 
26p. 

Reports - Evaluative/Feasibility (142) 
MF01/PC02 Plus Postage. 

^Computer Simulation; ^Construct Validity; Foreign 
Countries; ^Formative Evaluation; Goodness of Fit; 
^Mathematical Models; *Multitrait Multimethod 
Techniques; ^Research Methodology 
Academic Self Concept; ^Composite Direct Product 
Model (Browne); LISREL Computer Program; Violation of 
Assumptions 

ABSTRACT 

The multitrait-multimethod (MTMM) paradigm is used 
widely to assess construct validity, but D. A. Kenny and D. A, Kashy 
(1992) lament that even after 30 years we still do not know how to 
analyze MTMM data. The Composite Direct Product (CDP) model has 
recently attracted considerable attention. Its strengths and 
weaknesses are evaluated, and a stronger basis for testing the model 
is demonstrated. The CDP model is mathematically elegant and 
extremely parsimonious, but this parsimony is at the expense of very 
restrictive assumptions that undermine the formative evaluation of 
MTMM data. Furthermore, the CDP model provides an excellent fit to 
simulated data in which these assumptions are seriously violated, 
demonstrating that fit is not a sufficient test of these assumptions. 
Typically the CDP model is based on single indicators of each 
trait/method combination, but a stronger multiple-indicator version 
of the CDP model is demonstrated using new options available in the 
LISREL 8 computer program. Five tables present data illustrating the 
CDP model, using data from a study by B. M. Bryne and R. J. Shavelson 
(1986) on the relations among three academic self-concept traits. An 
appendix presents a LISREL setup for fitting the multiple indicator 
CDP model. (SLD) 
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Abstract 

The MTMM paradigm is used widely to assess construct validity, but Kenny and Kashy (1992) lamented that even after 
30 years we still do not know how to analyze MTMM data. The Composite Direct Product (CDP) model has recently 
attracted considerable attention. Here we evaluate its strengths and limitations, and demonstrate a stronger basis for 
testing the model. The CDP model is mathematically elegant and extremely parsimonious, but this parsimony is at the 
expense of very restrictive assumptions that undermine the formative evaluation of MTMM data. Furthermore, the CDP 
model provides an excellent fit to simulated data in which these assumptions are seriously violated, demonstrating that fit 
is not a sufficient test of these assumptions. Typically the CDP model is based on single indicators of each trait/method 
combination, but we demonstrate a stronger multiple-indicator version of the CDP model using new options available in 
L1SREL8. 
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Composite Direct Product Model 1 

Campbell and Fiske ( 1959) argued that construct validation requires both convergent and discriminant validity. 
They proposed the multitrait-multimethod (MTMM) design which is used widely to assess construct validity. In this 
design measures of two or more traits (e.g., school, reading, and mathematics self-concepts) are each measured with two 
or more methods (e.g., three different self-concept instruments). Traits are attributes such as abilities, attitudes, behaviors 
or personality characteristics, whereas methods refer broadly to multiple test forms, methods of assessment, raters, or 
occasions. Construct validity is supported when convergent validity (e.g., the extent of agreement among math self- 
concept scales from three instruments) and discriminant validity (e.g., the extent to which mathematics and verbal self- 
concepts can be distinguished) are high, and method effects (e.g., idiosyncratic variance associated with a method of 
measurement) are negligible. 

Although the original Campbell-Fiske guidelines are Jill used widely to evaluate MTMM data, important 
problems with the guidelines are well known (e.g., Althauser & Heberlein, 1970; Alwin, 1974; Campbell & O'Connell, 
1967; Marsh & Hocevar, 1983; Marsh, 1988; Schmitt & Stults, 1986; Sullivan & Feldman, 1979). Campbell and Fiske 
(1959) were aware of most of these limitations, specifically stating that their guidelines should be viewed as "common- 
sense desideratum" (p. 83). Their intent was to provide a systematic, formative evaluation of MTMM data at the level of 
the individual trait-method unit, qualified by the recognized limitations of their approach, not to provide summative, 
global summaries of convergent validity, discriminant validity and method effects. We recommend that these guidelines 
should be used as an initial formative evaluation of all MTMM data even when more sophisticated latent variable 
approaches are applied and that this formative orientation in the MTMM paradigm must not be lost in the development 
of mathematically more sophisticated approaches to MTMM data. More generally, Campbell and Fiske had a heuristic 
intention to encourage researchers to consider the concepts of convergent validity, discriminant validity, and method 
effects; in this intention they were remarkably successful. Unfortuately, (his formative, heuristic aspect that was the very 
essence of Campbell-Fiske approach seems to have been lost in the quest to develop ever more mathematically 
sophisticated approaches to MTMM data. Nevertheless, problems inherent in the application of the original Campbell- 
Fiske guidelines led to many alternative analytic approaches, including the composite direct product (CDP) model, and 
the new, multiple indicator CDP (MI-CDP) model that are demonstrated here. 

Composite Direct Product Model 

Most approaches to MTMM data implicitly assume that trait and method effects are additive, but Campbell and 
O'Connell (1982) suggested that the relation may be multiplicative. Both the additive and multiplicative models posit 
that correlations between traits measured with the same method will be higher than correlations between traits measured 
with different methods - a method effect. If this method effect is additive, then the increase in correlation due to this 
method effect is expected to be of a similar size for large and small correlations. In a multiplicative model, however, the 
method effects are systematically larger for traits that are more highly correlated, systematically smaller for traits that are 
less correlated, and zero for traits that are uncorrelated (i.e., the method effect multiplied by zero is zero). Campbell and 
O'Connell (1967, 1982) offered two different interpretations of this multiplicative effect. The differential augmentation 
perspective is that observed correlations are a multiplicative function of the true correlation and a method bias. 
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Composite Direct Product Model 2 

According to this perspective, when true traits are uncorrelated there will be no bias (i.e., the method effect multiplied by 
zero is zero). In contrast, when traits are substantially correlated the correlation between the traits based on the same 
method will be biased so long as the method effect is nonzero. This portrayal of method effects differs from a purely 
additive model that implicitly assumes that the size of method effects does not vary according the size of true trait 
correlations. 1 The differential attenuation perspective suggests that the use of different methods will attenuate the true 
correlation between two traits. The extent of this attenuation, however, will vary according to the size of the con-elation. 
If the true trait correlation is already zero, the correlation cannot be attenuated. In contrast, if the true trait correlation is 
substantial, then the empirical correlation can be attenuated substantially. According to this perspective, the correlation 
between two traits measured by the same method is a more accurate estimate of the true correlation, and this correlation 
is attenuated when different methods are used. This perspective is apparently consistent with the typical simplex pattern 
of relations observed in longitudinal data whereby the size of correlations between traits declines systematically as the 
time between the collection of the measures becomes longer. 

Browne (1984), based in part on earlier work by Swain (1975), described the composite direct product (CDP) 
model and developed the MUTMUM statistical package (Browne, 1990). The CDP model posits a multiplicative rather 
than an additive combination of latent trait and method correlations. According to the CDP model there are two 
component correlation matrices in a MTMM matrix of correlations among latent variable scores (Pc), one containing 
correlations between latent traits (Pt) and the other containing correlations between latent methods (Pm). The covariance 
matrix of measured variables with dimension (mt x mt) can be expressed as: 

(1) S = D (PmxPt +E) D 

where Pm is an (m x m) latent variable score correlation matrix of method components, Pt is a (t x t) latent variable score 
correlation matrix of trait components, D is a (mt x mt) positive definite, diagonal matrix of scale constraints reflecting 
latent variable score standard deviations, E is a positive definite diagonal matrix of uniquenesses reflecting the ratio of 
unique score variance to latent variable score variance, and x indicates the right direct (Kronecker) product of Pm and Pt. 
The values of D are typically of no interest and are designed primarily to absorb scaling changes similar to those 
involved in going from a covariance matrix to a correlation matrix. The E values, however, represent the ratios of unique 
score variances to latent variable score variances. Browne (1984, 1989) noted that these values can be interpreted as the 
correlation between an observed and latent variable score, an "index of communality," when transformed by the formula: 

(2) communality (Ti, Mr) » 1 / [1 + (E(Ti, Mr)] 

According to the CDP model, the correlation matrix Pc, appropriately corrected for attenuation, has the direct product 
structure: 

(3) Pc = Pm x Pt 

where Pm has typical element r(Mr, Ms) and Pt has typical element being r(Ti, Tj). From this definition it follows that 
for latent variable scores 

(4) r(TiMr, TjMs) = r(Ti,Tj) r(Mr,Ms) « (Tij x Mrs) 
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It is useful to demonstrate the relations among Pm, Pt, and Pc for a MTMM design with 3 traits and 3 methods like that 

considered here (see Table 1). All elements of Pt are multiplied by each element of Pm. Thus, the correlation between 

traits 1 and 2 measured with method 1 is T21 multiplied by Ml 1 = 1 so that the product is simply T21 . Similarly, the 

correlation between trait 1 measured with methods 1 and 2 (i.e., a convergent validity) is M21 times Tl 1 = 1 so that the 

convergent validity is simply M2 1 . Thus, the coefficients in the off-diagonal of Pm reflect convergent validity. Note also, 

that the correlation between the same traits is assumed to be constant across all methods (i.e., r(TlMl,T2Ml) = 

r(TlM2,T2M2) = r(TlM3,T2M3) = T21). Similarly, the correlation between two methods - convergent validity - is 

assumed to be the same across all traits (i.e., r(TlMl,TlM2) = r(T2Ml,T2M2) = r(T3Ml,T3M2) = M21). Because the 

36 off-diagonal values in Pc are expressed in terms of only 6 estimated parameters (T21, T13, T23, M21, M31, M32), the 

CDP model is very parsimo nious. 

Insert Table 1 About Here 

Relation to Campbell-Fiske Guidelines 

Browne (1984, 1989; also see Bagozzi & Yi, 1990; Cudeck, 1988; Goffin & Jackson, 1992) notes that an 
important advantage of this model is that it provides parameter estimates that can be used to evaluate the original 4 
Campbell-Fiske guidelines. 

1. Convergent validities arc substantial [r(TiMp, TiMs) » 0]. In the CDP model r(TiMp, TiMq) = r(Ti,Ti) 
r(Mp,Mq) = r(Mp,Mq) » 0 so that, each convergent validity is equal to one of the off-diagonal values in Pm. Hence, 
this criterion is satisfied whenever all the off-diagonal values in Pm are statistically significant, large, and positive. 

2. Convergent validities are higher than heterotrait-heteromethod (HTHM) correlations [r(TiMp, TiMq) > 
r(TjMp, TiMq) and r(TiMp, TiMq) > r(TjMq, TiMp)]. In the CDP model r(TiMp, TiMq) > r(TiMp, TjMq) implies 
[r(TiMp, TjMq)] / [r(TiMp, TiMq) ] = [r(Ti,Tj) r(Mp,Mq)] / [r(Mp, Mq) ] = r(Ti, Tj) < 1.0. Hence, the latent variable 
trait correlations, the off-diagonal values in Pi, arc the ratio of HTHM correlations to the convergent validities. Hence, 
this criterion is met whenever the off-diagonal values of Pt are less than 1.0. This will always be the case so long as the 
CDP solution is proper (i.e., Pt is positive definite), although stronger support may require the off-diagonals to be 
significantly or substantively »es.s than 1.0. 

3. Convergent validities are higher than heterotrait-monomethod (HTMM) correlations [r(TiMp, TiMq) > r 
(TiMp, TjMp) and r (TiMp, TiMq) > r (TiMq, TjMq)]. In the CDP model r(TiMp, TiMq) > r(TiMp, TjMp) implies that 
[r(TiMp, TjMp)] / [r(TiMp, TiMq) ] = [r(Ti,Tj) ] / [r(Mp, Mq) ] < 1.0. According to the CDP model, the ratio of HTMM 
correlations to the convergent validities is the ratio of trait correlations to method correlations. Hence this criterion is met 
when all the off-diagonal values in Pt are less than all the off-diagonal values in Pm. 

4. The pattern of correlations among different traits is similar for different methods [r (TiMp, TjMq) > r (TkMp, 
TiMq) implies r (TiMr, TjMs) > r (TkMr, TIMs)]. In the CDP model, this criterion is met whenever the CDP model fits 
the data because: r(TiMr, TjMs) / r(TkMr, TIMs) = r(Ti, Tj)/ r(Tk, Tl) has the same value for any Mr and Ms. 

Method effects - an typically undesirable source of bias - are inferred when HTMM correlations substantially 
exceed HTHM correlations [r(TiMp, TjMp) > r (TiMp, TjMq)]. Campbell and Fiske (1959, p. 85) stated that "the 
presence of method variance is indicated by the difference in level of correlation between parallel values of the 
monomethod block and the hctcromcthod block, assuming comparable reliabilities among the tests" and March (1988) 
opcrationalized this test of method effects. In the CDP model, r(TiMi\ TjMr) > r(TiMr, TjMs) implies [r(TiMr, TjMs)] / 
[r(TiMr, TjMr)] = [r(Ti,Tj) r(Mr, Ms) ] / [r(Ti,Tj) r(Mr,Mr)] = [r(Ti,Tj) r(Mr, Ms)]/ [r(Ti,Tj)] = r(Mr, Ms) < 1.0. Hence, 
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according to (he CDP model, there are method effects whenever the correlations in Pm are less than 1, and so there are 
always method effects when the CDP results in a proper solution. Also note that r(Mr, Ms) reflects both convergent 
validity (see guideline 1) and method effects. Whereas this observation appears paradoxical from the traditional 
"additive" perspective, it follows logically from the "multiplicative" perspective underlying the CDP model. 

It is also possible to place further constraints on the CDP model that may be useful in particular situations. 
Thus, for example, it is possible to further restrict the structure of E, the diagonal matrix of uniquenesses, so that it also 
has a direct product structure (Browne, 1984, 1989; Wothke & Browne, 1990). This constraint has the effect of creating 
trait communality indices that have the same rank order for each method (e.g., if math self-concept is most reliable for 
one instrument it is most reliable for all instruments) and of producing method c jmmunality estimates that have the same 
Kink order for each trait. This additional constraint is most reasonable for multiple battery data where, for example, the 
same instrument is administered to different individuals or the same individual on different occasions. Also, when the 
general CDP model does not result in proper solution, it is possible that the more restrictive solution with a direct product 
error structure will be proper. However, the direct product error structure is not central to interpretations of convergent 
validity, discriminant validity, and method effects that is the focus of the present investigation (for further information 
see Browne, 1984; 1989). If the covariance matrix rather than the con-elation matrix is analyzed, it is possible to further 
restrict the structure of D, the diagonal matrix of scale constraints reflecting latent variable score standard deviations. 
Proper Solu tions. Parsimony, and Goodness of Fit. 

An important advantage of the CDP model is that, based on the relatively limited number of published 
applications (e.g., Bagozzi, 1992; in press; Bagozzi & Yi, 1990; 1991; 1992; Bagozzi, Yi, & Phillips, 1991; Browne, 
1984* 1989; 1990; Cudeck, 1988; Goffin & Jackson, 1992; Kumar & Dillon, 1992; Lastovicka, Murry & Joachimslhaler, 
1990; Wothke & Browne, 1990) it typically results in proper a solutions that is very parsimonious and is able to fit 
MTMM data at least reasonably well. Marsh and Grayson (in press) further noted that because of its parsimony, it may 
be appropriate to f»r the CDP model in MTMM designs in which the more widely used confirmatory factor analysis 
(CFA) approach is not typically recommended (e.g., studies with small Ns, only two traits, or only two methods). Also, 
Fiske and Campbell (1992) have indicated that a number of different researchers are now optimistic about the CDP 
model. 

Whereas a general discussion of goodness of fit and related issues is beyond the scope of this article (sec, 
Bcntler, 1990; Cudeck & Henly, 1991; Marsh, Balla & McDonald, 1988; McDonald & Marsh, 1990), there are some 
specific issues that are particularly important for the evaluation of the CDP model. Because the CDP model is so 
parsimonious relative to most latent-trait MTMM models, the comparison of goodness of fit with other approaches will 
vary substantially depending on particular index that is used. Thus, for example, indices that control for model 
parsimony (e.g., the parsimony indices described by Muiaik, et. al, 1988 or the Tucker-Lewis Index as noted by 
McDonald & Marsh, 1990) or for capitalizing on chance (e.g., the single-sample cross-validation index described by 
Browne & Cudeck, 1989) arc more likely to favor more parsimonious models like the CDP than indices like the X 2 
value or the relative noncentrality index (RNI; McDonald & Marsh, 1990) that do not control for the these 
characteristics. Apparently, there is also some confusion in the literature as to what constitutes a "proper" CDP solution. 
We interpret the CDP solution to be proper when Pc, Pt, D, and E arc all strictly positive definite. In a recent comparison 
of the CDP model with other latent-variable approaches to MTMM data, however, Goffin and Jackson (1992) claimed 
several solutions to be "proper" even though the Pt or E were not positive definite. 
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Goffin and Jackson (1992) fit the CDP model with Browne's MUTMUM (1990) statistical package that is 
specifically designed for this purpose. It is much easier to fit the CDP model with MUTMUM than with LISREL 7 (e.g., 
Wothke & Browne, 1990), and MUTMUM offers considerable advantages (e.g., appropriate standard errors and 
automatic computation of reliability estimates). Also, MUTMUM automatically imposes inequality constraints on 
parameter estimates in Pc, Pt, E and D that "result in maximum likelihood estimates in some situations where the 
likelihood function would otherwise be unbounded" (Browne, 1990, p. 4). However, these inequality constraints are not 
sufficient to ensure that Pt and Pm are positive definite (Browne, 1990, p. 4), and MUTMUM does not report whether the 
Pt and Pm are positive definite. Thus, for example, data considered by Goffin and Jackson resulted in a Pt matrix that 
was not positive definite even though none of the correlations were greater than 1.0, and they mistakenly claimed that 
this solution was proper. Hence, MUTMUM users need to evaluate the positive definiteness of Pt and Pm. 

MUTMUM also restricts negative variance estimates to be non-negative. Typically this results in the offending 
parameter taking on a zero value that is on the boundary of the constrained parameter space and in a slight decrement in 
goodness of fit reflecting this implicit inequality constraint. Thus, for example, when values of the E matrix are zero, the 
corresponding communality estimate is 1.0, and this is referred to as a "boundary condition" in the MUTMUM output. 
Joreskog and Sorbom (1989) emphatically stated that "it should be emphasized that constraining error variances to be 
non-negative does not really solve the problem. Zero estimates of error variances arc as unacceptable as are negative 
estimates" (p. 215). In the context of MTMM data. Marsh (1989) argued that whereas such rcparamctcrizations may be 
useful in some situations, solutions with a zero variance estimate arc still improper and should be treated with the same 
caution as if the parameter estimate were negative. Hence, we disagree with the claim by Goffin and Jackson that their 
solutions are "proper" even though they contain boundary conditions in which variance estimates thai would otherwise 
be negative are constrained to be zero. 
Demonstrat ion of the CDP Model For the Bvrne Data. 

Byrne and Shavclson (1986; also see Marsh, 1988; 1989) examined the relations among three academic self- 
concept traits (Math, Verbal, and General School) measured by three different instruments (sec Table 2). The 9 scores 
representing all combinations of the 3 traits and 3 methods were based on multi-item scales and the three instruments had 
strong psychometric properties. Consistent with a priori predictions, they found that the Math and Verbal self-concepts 
were nearly uncorrclatcd with each other and were substantially correlated with School self-concept. Marsh (1989) noted 
that this "is an exemplary MTMM study because of the clear support for the Campbcll-Fiskc guidelines, the large sample 
size (817, after deleting persons with missing data), the good psychometric properties of the measures, and the a priori 
knowledge of the trait factor structure" (p. 348). Also, the predicted lack of correlation between Math and Verbal self- 
concept satisfies the Campbell and Fiske recommendation to include two traits "which are postulated to be independent 

of each other" (p. 104). Th ere is good support for the Campbell-Fiske guidelines in that: 

Insert Tabic 2 About Here 

1. Convergent validities are substantial, varying between .54 and .87 (mean r=.70); 

2. Convergent validities are higher than HTHM correlations (mean r=.31) in all 36 comparisons; 

3. Convergent validities arc higher than HTMM correlations (mean r=.35) for 33 of 36 comparisons. [All three 
failures involve M3 where correlations among the traits (mean r « .44) are higher than for Ml (.28) or M2 (.33), 
suggesting moderate method effects associated with at least M3J. 
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4. The pattern of correlations among different traits is similar for different methods in that correlations between 
T2 (Reading self-concept) and T3 (Math self-concept) arc consistently small (mean r=.06) whereas Tl (General 
Academic self-concept) is substantially correlated with both T2 (mean r=.42) and T3 (mean r^.45). 
Results and Discussion of the CDP Model With the Bvrne Data. 

The CDP model resulted in what we interpreted to be an improper solution in that parameter estimates from the 
E diagonal matrix included one estimate of 0 - a boundary condition (see Table 3). Hence, even though this model 
provided a good fit to the data (RNI = .966, X 2 (21, N=817)= 201), we chose to interpret this model cautiously. As a 
pragmatic alternative, we fit the CDP model in which the errors are posited to be structured as direct products (sec 
Browne, 1984; Cudeck 1988). This resulted in a fully proper solution and a somewhat poorer fit to the data (RNI = .956, 
X 2 (25, N=817)= 262). For both solutions, the Pm correlations (Table 3) are consistently very large and consistently 
larger than the Pt correlations, whereas the Pt correlations are substantially smaller than 1. This implies clear support for 
all the Campbell-Fiske guidelines and strong support for the construct validity of these measures. The relative lack of 
correlation between T2 and T3 observed in the MTMM matrix and predicted a priori is evident in Pt. Similarly, the 
apparently stronger agree ment between measures based on M 1 and M2 is evident in Pm, 

Insert Table 3 About Here 

The CDP model offers a mathematically elegant and parsimonious model of MTMM data. Subject to the 
continued demonstration of its success, we recommend that the CDP model should be used in MTMM studies. We do, 
however, have some reservations. In particular, its parsimony is achieved at the expense of implicit assumptions that we 
find worrisome. 

1. In the CDP model, the convergent validities for all the different traits arc necessarily equal (i.e., r(TiMl, 
TiM2) = r(Ml, M2) for ail values of i). In the Byrne data, however, the convergent validities are consistently larger for 
T3 (mathematics self-concept) than Tl or T2. In many MTMM studies this restriction would be unreasonable. 
Furthermore, imposing these constraints may undermine the formative value of the MTMM design, since a major focus 
of many studies is to compare convergent validities for different traits. 

2. In the CDP model, the size of method effects is necessarily the same for different traits (i.e., 
r(TiMr,TjMs)/r(TiMr,TjMr) = r(Ti,Tj) x r(Mr,Ms)/ r(Ti,Tj) = r (Mr,Ms) for all values of i and j). In the Byrne data, 
however, there appear to be stronger method effects associated with M3 than with Ml or M2. Again, this restriction 
would be unreasonable in many MTMM studies and may detract from the formative value of the procedure. The focus of 
many MTMM studies is to compare the extent of method effects associated with different methods of measurement, 

3. In the CDP model, the size of correlations among traits is necessarily the same for all methods (i.e., 
r(TiMr,TjMr) = r(Ti,Tj) for all values of r). In the Byrne data, however, correlations among traits arc systematically 
lower for Ml and systematically higher for M3. It is also possible that Ti and Tj arc the most highly correlated traits for 
Ml, but are the least highly correlated for M2, but the CDP model does not allow for this possibility. Again, this 
restriction would be unreasonable in many MTMM studies. 

In the CDP model, Pt correlations typically reflect the pattern of correlations among traits in the MTMM matrix, 
but only if this pattern is consistent across methods. Pm correlations typicalty reflect the extent of agreement between 
different methods, but only if the agreement is consistent across all traits. Whereas the overall fit of the model provides 
an indirect test of these assumptions, overall fit indices do not provide information that is particularly useful in 
evaluating the specific assumptions. Furthermore, there is an implicit assumption that these assumptions are necessarily 
satisfied when the overall fit is good, but in the next section we question the validity of this assumption. Common sense 
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suggests that these assumptions will typically be false so that a more detailed evaluation of the implications of violating 
these assumptions is needed in actual applications of the CDP model. Finally, because of these implicit invariancc 
constraints, the CDP model does not provide a very useful formative evaluation of specific trait-method units which is, 
perhaps, the primary purpose of the MTMM design. 
Results and Discussion nf t he CDP Model With Artificial Data. 

In the CDP model, researchers must rely heavily on the overall goodness of the model to evaluate the 
reasonableness of a host of very restrictive assumptions. It is clear that in at least some perhaps most ~ MTMM studies 
these assumptions arc not reasonable. Yet, our informal reading of published applications of the CDP model (e.g., Bagozzi, 
1992; in press; Bagozzi & Yi, 1990; 1991; 1992; Bagozzi, Yi, & Phillips, 1991; Browne, 1984; 1989; 1990; Cudcck, 1988; 
Goffin & Jackson, 1992; Kumar & Dillon, 1992; Lastovicka, Murry & Joachimsthaler, 1990; Wothkc & Browne, 1990) 
has not revealed any situations in which the CDP model resulted in a particularly poor fit to the data. This may call into 
question the falsifiability of the CDP model or, alternatively, may merely reflect biases in the data used to illustrate the 
CDP model. 

In order to illustrate our concerns more clearly, we constructed an artificial MTMM matrix that seriously violates 
CDP assumptions but is easily evaluated with the Campbell and Fiskc (1959) guidelines. Wc generated the data from a 
CFA population model (see Table 4) in which thene were small method effects [samll method factor landings], small to 
moderate trait correlations, substantial trait variance for Tl and T2 [large trait factor loadings], and only weak trait 
variance for T3. Inspection of the resulting (simulated) MTMM matrix (Table 4), consistent with the design of the data, 
indicates good support for the convergent and discriminant validity of Tl and T2, but little or no support for cither the 
convergent or discriminant validity of T3. This MTMM matrix should be particularly troublesome for the CDP model 
because the CDP model requires that all convergent validities associated with a given method are equal whereas the 
convergent validity of T3 is weak and much weaker than that of T2 and T3. Based on a hypothetical N=500, however, the 
CDP model provided a remarkably good fit to this artificial data (RNI = 1.01 1, X 2 (21) = 1 1.15). Parameter estimates for 
the CDP model (Table 4) reflect the pattern of trait correlations with reasonable accuracy but not the large differences in 
convergent validities for the three traits. Furthermore, the Pm correlations -- the convergent validities -- {ire all greater than 
.9. These appear to be grossly inflated in relation to the actual population model used to generate the data - particularly 

for T3 that had very weak convergent validity. 

Insert Table 4 About Here 

These results provide one example in which the CDP model provides an apparently excellent fit to the data even 
though assumptions underlying the CDP model were seriously violated. A comparison of the parameter estimates with the 
population model, however, revealed that the CDP model seriously misrepresented the MTMM data even though it 
provided such a good fit to the data. This example, even more than the results of the Byrne data, demonstrates that it is 
important to critically evaluate parameter estimates based on the CDP solution. Overall goodness of fit tests apparently do 
not always provide an adequate basis for evaluating the assumptions underlying the CDP model and can be very 
misleading. This can be very dangerous. 

Multiple Indicato r Composite Direct Product Models 

The heuristic value of the original Campbell-Fiske guidelines is widely recognized, but their application is 
criticized in particular for issues related to their reliance on correlations among measured variables instead of latent 
constructs. Ironically, latent variable models -- including the CDP model emphasized here - typically begin with single 
indicators of each trait/method combination even though each trait/method combination is often based on multiple 
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indicators (e.g., the individual items in the self-concept scales used for (he Byrne data). However, incorporating multiple 
indicators of each trait/method combination into the latent variable approaches has important advantages (also see 

Marsh, in press; Marsh & Hoccvar, 1988;. 

Insert Table 5 About Here 

1. Implicit in all latent-variable MTMM models based on relations among scale scores is the assumption that 
the researcher's a priori structure (i.e., the one implied by how scores are combined) accurately reflects the true factor 
structure. In particular, it is implicitly assumed that the multiple indicators can be represented by a single common factor. 
Unless there is empirical support for this a priori factor structure, however, the interpretation of the MTMM results is 
problematic. If multiple indicators designed to measure the same scale actually reflect different traits, or multiple 
indicators from different scales actually reflect the same trait, then scale scores cannot be interpreted in terms of trail and 
method effects. Indeed, it is ironic that latent-variable approaches to MTMM analyses suffer from this problem even 
when multiple indicators of each scale arc collected. With the multiple indicator approach demonstrated here, the 
evaluation of the typical first-order model (e.g., Table 5) provides an explicit test of this a priori model that is implicit 
and untestable in latent variable approaches based on scale scores. An evaluation of the fit of the first-order model 
provides a test of the implicit assumption that the multiple indicators associated with a particular trait/method 
combination reflect a single, well-defined, common factor. If this first-order model is unable to fit the data reasonably 
well, then it may be necessary to include additional parameters (e.g., correlated uniqueness - particularly for multiple 
battery data) that may alter the pattern of relations among the latent constructs. If no theoretically defensible first-order 
model is able to fit the data, further inferences about trait and method effects arc dubious no matter what approach is 
used. A major problem in MTMM studies is the use of instruments with weak psychometric properties. Whereas the 
multiple indicator approach advocated here may not solve this problem, its application will more clearly identify the 
source of the problem. This initial step contributes substantially to the formative evaluation of MTMM data at the level 
of each trait/method combination which is a primary puipose (he MTMM design. 

2. In the first-order model using the multiple indicator approach, the MTMM matrix consists of correlations 
among latent constructs that arc appropriately corrected for measurement error. These latent constructs can be directly 
evaluated with the traditional Campbcll-Fiskc guidelines and many criticisms of the guidelines are met by this multiple- 
indicator procedure. Whereas single-indicator latent variable approaches arc designed to provide information more or 
less relevant to the Campbeil-Fiskc guidelines, the guidelines themselves are applied directly in the multiple indicator 
approach advocated here. It is clearly better to apply the Campbell and Fiske (1959) guidelines to a MTMM matrix of 
correlations among latent variables than to a MTMM matrix of correlations among observed variables. 

3. In latent variable models that begin with scale scores, reliability estimates are based on communality 
estimates. A scale is inferred to lack reliability when it is not substantially correlated with at least some of the other 
scales in the MTMM design. This approach, however, confounds a lack of correlation due to unreliability in the scale 
scores and a lack of correlation among the latent constructs. In the multiple indicator approach, however, unreliability in 
the scales is estimated separately from correlations among the latent constructs. The ability to separate these two sources 
of information is particularly important in the formative evaluation of the different measures and their construct validity. 

4. Particularly in multiple battery studies in which the multiple methods are multiple occasions or multiple 
raters responding to the same materials, correlated specific variances are likely to positively bias estimates of convergent 
validities and inferred trait effects. When the multiple methods are different occasions, (he problem of autocorrelated 
specific variances is well known. Joreskog (1979), for exai^plc, noted that if the same measurements are used on 
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multiple occasions, the corresponding error variables will tend to correlate, and in order to get accurate estimates of 
relations among the constructs, correlations among errors must be included in the model. The problem is also relevant, 
however, when the same indicators arc given to multiple raters (e.g., when employees evaluate their own work 
performance, arc evaluated by peers, and are evaluated by superiors using the sdmc multitrait rating instrument). In the 
typical latent variable approach that begins with scale scores, however, this potentially serious source of bias cannot be 
tested and there is no way to control the bias if it docs exist. Using the multiple indicator approach, however, researchers 
arc able to test for the existence of such correlated errors and their inclusion in the model controls for this source of bias 
in inferences about trait and method effects. 

5. The multiple indicator approach actually results in a MTMM matrix of correlations among latent constructs 
and so it can be extended to incorporate any latent variable model that is based on MTMM correlations. Furthermore, all 
latent variable models based on the multiple indicator approach are nested under the corresponding first-order model so 
that the first-order model provides the optimal fit lor the more parsimonious latent variable models. In this sense, the 
Iv nt variable model is merely trying to explain covariation among latent constructs in the first-order model with a more 
parsimonious structure. Thus, using the multiple indicator approach, the first-order model provides an important basis for 
evaluating latent variables models. If the goodness of fit of the latent MTMM model is substantially poorer than that of 
the first-order model or if interpretations based on the latent MTMMhiodel differ substantially from those based on the 
first-order model, then the latent variable model should be rejected \q favor of the first-order model. Marsh (in press; 
Marsh & Hoccvar, 1988) applied this multiple indicator approach tolSFA models from the taxonomies proposed by 
Widaman (1985) and Marsh (1989). f * 

Marsh (in press) als'Miotcd that the multiple indicator approach could be generalized to other latent variable 
approaches such as the CDP model described here. Because there arrfapparen(jy no published demonstrations of this 
proposal, we illustrate the multiple indicator CDP (MI-CDP) model wttfo Ihc Byrnc data considered earlier. Whereas this 
model can be lit with several different statistical packages, it apparcnl^pannot be fit with the MUTMUM package that 
is specifically designed for the single-indicator approach to the CDP m\jdpl. However, the type of constraints required to 
fit the CDP model (i.e., requiring one parameter to be the product of tw<o pthCi*paramcter estimates) is greatly facilitated 
by new options available in LISREL 8 (the LISREL 8 set-up is summarised- in Appendix 1). Hence, a secondary puipose 
of this demonstration is to illustrate this new option in LISREL. 
Results of tWe Multiple In dicator Approach 

For purposes of this analysis, three-item parcels were constructed io reflect each of 9 trait/method combinations 
in the Byrne data - a total of 27 measured variables. Thus, for example, thcritpm? used to infer School self-concept (Tl) 
with the first self-concept instalment (Ml) were divided into three item-parcels to form three indicators of this 
trait/method combination. A critical first step in this analysis is to fit a typical first-order CFA model in which the 27x27 
covariancc matrix is fit to a first -order factor model. This first-order model is well-defined (Table 5) in that the solution 
is proper, each of the 27 measured variables loads substantially on its firjt-ordcr factor, and the goodncss-of-fit is good 
(RNI = .939) even though the X 2 (288, n=817) = 1506 is substantial. 

Of particular relevance to the present application is the 9 x 9 matrix of correlations among the first order factors 
(see Table 5). Whereas this latent MTMM matrix is like a typical manifest MTMM matrix based on scale scores that is 
used to assess the Campbell-Fiske guidelines, there arc important differences. In particular, the correlations in this latent 
MTMM matrix are correlations among latent constructs that have been appropriately corrected for measurement error. 
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Hence, the Campbell-Fiske guidelines can be applied directly to this latent MTMM matrix and mary of the objections to 
the application of these guidelines to correlations among scale scores are overcome. 

In the present application, because of the psychometric strength of the scales used in the Byrne data, the 
interpretation of the guidelines to the latent MTMM matrix of correlations among latent variables is reasonably similar to 
that described earlier for the manifest MTMM matrix based on scale scores. Support for the Campbell-Fiske guidelines 
applied to the latent MTMM matrix (Table 5) is, however, somewhat stronger. The mean convergent validity is higher 
for the latent MTMM matrix than the manifest MTMM matrix (.80 vs. .70), even though the mean HTHM correlation 
(.31 vs. .31) and the mean HTMM correlation (.37 vs. .35) differ only slightly. Consistent with these results, Campbell 
and Fiske's second and third guidelines were satisfied in 69 of 72 comparisons for the manifest MTMM matrix but are 
satisfied for all 72 comparisons for the latent MTMM matrix. Inspection of the parameter estimates for the latent MTMM 
matrix reveals trends similar to those observed earlier in the manifest MTMM matrix: (a) convergent validities tend to be 
higher for T3 (mean r = .90) than for Tl (mean r = .74) or T2 (mean r = .75), (b) convergent valdities relating Ml and 
M2 tend to be higher than those associated with M3, (c) within each method, Tl is substantially correlated with T2 
(mean r = .50) and T3 (mean r = .56), but T2 and T3 are nearly uncorrected (mean r = .07); and (d) correlations among 
traits are systematically higher for M3 (mean r = .48) than for Ml (mean r = .30) or M2 (mer\ = .31), suggesting that 
there is more method effect associated with M3. We suspect, that differences in interpretations in the latent and manifest 
MTMM matrices will be larger in other studies in which the psychometric properties of the responses are not so strong as 
in the Byrne data. 

Using the multiple indicator approach, the CDP model merely posits a more restrictive pattern of correlations 
among the first-order latent constructs such that Pc = Pm x Pt (sec earlier discussion of Table 2). Thus, the 36 
correlations in the 9 x 9 correlation matrix can be represented by only 6 parameter estimates - the 3 correlations in Pm 
and the 3 correlations in Pt. All correlations in the MTMM matrix are either equal to one of these 6 parameter estimates 
or are the product of two of these parameter estimates. With new options available in LISREL 8, this pattern can be 
represented directly as constraints on the estimated correlations (see Appendix 1). In the multiple indicator approach in 
which the MTMM matrix represents correlations among latent constructs, there is no need to posit additional parameters 
associated with the E and D diagonal matrices in the typical CDP model. This is a very important advantage because it 
separates issues associated with the hypothesized structure of trait and method effects from those associated with 
measurement issues. 

The MI-CDP model is well-defined (Table 5) in that the solution is proper, each of the 27 measured variables 
loads substantially on its first-order factor, and the goodness-of-fit is reasonably good (RNI = .911) even though the X 2 
(318, n=817) = 2100 is substantial. Because the MI-CDP model is nested under the first -order model, the comparison of 
the two models is facilitated. The change in X 2 (2100 - 1506 = 594) seems large relative to the change in df (318 - 288 = 
30), as docs the change in RNIs (.939 vs .911). Whereas the fit of the MI-CDP model appears to be reasonable, the 
overall fit statistics should be interpreted cautiously. Much of the good fit due to the MI-CDP model is due to the fact 
that much of the covariation among the measured variables can be explained in terms of the first-order factors. Thus, for 
example, a first-order model in which all 9 latent constructs are posiicd to be uncorrelated (i.e., the factor correlation 
matrix is an identity matrix) is able to explain a substantial amount of variation ((RNI = .644, X 2 (324, n=8 17) = 6886). 
Whereas the MI-CDP model is obviously able to fit the data much better than this model, the comparison of these three 
models demonstrates the much of the ability of the MI-CDP model to fit the data has nothing to do with the posited 
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pattern of correlations among the latent MTMM variables (see further discussion of the "target coefficient" proposed by 
Marsh, in press; and Marsh & Hocevar, 1988). 

A comparison of the parameter estimates (Table 5) based on the first-order model and the MI-CDP model is, 
perhaps, even more critical than the comparison of the ability of each model to fit the data. Whereas the first-order factor 
loadings and measured variable uniquenesses are reasonably similar for the two solutions, the factor correlations that are 
of critical concern differ substantially. In comparing the two models, it is important to recall that the MI-CDP is nested 
under the first-order model. In this sense, the MI-CDP is merely trying to approximate the parameter estimates in the 
first-order model with a more parsimonious model. Hence, if the factor correlations in the first-order model do not 
reflect a direct product structure, then the MI-CDP model is inappropriate. An inspection of the factor correlations in the 
MI-CDP solution suggests that the imposition of the direct product structure seriously distorts the pattern of correlations. 
The mean convergent validity is approximately .80 across all traits in both solutions. By definition, however, the mean 
convergent validity is the same for Tl, T2, and T3 (.80) in the MI-CDP solution, even though the convergent validities 
are much higher for T3 (mean r = .90) than for Tl (mean r = .75) or T2 (mean r = .74). Also by definition, the average 
correlation ofthe three traits within each method is the same for each method in the MI-CDP model, even though 
correlations among traits are larger for M3 (mean r = .48) than for Ml (mean r = .30) or M2 (mean r = .31). Finally, the 
relative size of correlations among the three traits differs in the two models. Whereas T2 (reading self-concept) and T3 
(math self-concept) were posited to be nearly uncorreiated and found to be nearly uncorrelated in the first-order model 
(mean r = .07), the correlation between T2 and T3 is substantially larger (r = .23 within each method) for the MI-CDP 
model. Also, whereas Tl (general school self-concept) was posited to be substantially correlated with T2 and T3 and 
found to be substantially correlated in the first-order model (mean rs of .50 and .56), these correlations were substantially 
smaller in the MI-CDP model (r = .37 for both correlations within all three methods). This inspection of parameter 
estimates, even more than the evaluation of goodness-of-fit, suggests that the MI-CDP model is not able to explain 
adequately the Byrne data. These results led us to reject the CDP model as an appropriate representation of the Byrne 
data. 

A Comparison of the Single and Multiple Indicator Approaches 

A basic supposition of this article is that the multiple indicator approach to MTMM data is always superior to 
approaches that begin with a single indicator of each trait/method combination. There are important advantages to the 
multiple indicator approach that are particularly important for the CDP model. The a ability to explicitly test the 
underlying factor structure and to apply the Campbell-Fiske guidelines directly to correlations among latent variables are 
sufficiently important advantages of the multiple indicator approach to justify its consideration. Particularly in the CDP 
single-indicator approach, measurement error is determined so as to optimize support for the assumptions underlying the 
CDP model. Hence, measurement issues may be confounded with tests of the structure of trait and method effects in the 
single-indicator CDP model. In the MI-CDP model, however, these aspects of the model are separated. Furthermore, in 
multiple battery data that may be most appropriate for the CDP model, there are likely to be correlated errors that cannot 
be easily tested or controlled with the single-indicator approach. We suspect that the advantages of the MI-CDP approach 
will be even greater than observed with the Byrne data in studies where the psychometric properties of the scales are not 
so strong or multiple battery data is considered. 

For present purposes the single-indicator and multiple-indicator approaches have been treated as different 
approaches to MTMM data, but this is overly simplistic. Ultimately, the single-indicator approach must be viewed as a 
special case of the more general multiple-indicator approach in which many implicit, usually untested, and perhaps 
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arbitrary restrictions about the structure of the multiple indicators are imposed (e.g., that the multiple indicators of each 
trait/method combination reflect a single, well-defined, common factor). The implications of these implicit assumptions 
cannot be easily tested with the single indicator approach but can be tested with the multiple indicator approach. Hence, 
the critical issue in the comparison of the multiple-indicator and single-indicator approaches is the usefulness of the 
restrictions imposed in the single indicator approach. Support for the single indicator approach must be based on logical 
arguments and empirical demonstrations that such restrictions are justified, and these tests require the application of the 
multiple indicator approach recommended here. 

Summary and Implications 

Although the CDP model has not been applied as widely as other approaches to MTMM data, particularly the 
CFA approach, results (e.g., Bagozzi, 1992; in press; Bagozzi & Yi, 1990; 1991; 1992; Bagozzi, Yi, & Phillips, 1991; 
Browne, 1984; 1989; 1990; Cudeck, 1988; Goffin & Jackson, 1992; Kumar & Dillon, 1992; Lastovicka, Murry & 
Joachimsthaler, 1990; Wothke & Browne, 1990) have suggested that it typically results in proper solutions and provides 
an apparently good fit to MTMM data. Consistent with Browne's claim, the CDP model provides evidence about the 
Campbell-Fiske guidelines and about convergent and discriminant validity as embodied in these guidelines. It it also 
claimed that the CDP model provides parameter estimates that are typically consistent with those observed in the 
MTMM matrices. However, the results of the present investigation demonstrate some limitations in the usefulness of 
particularly the single-indicator CDP model. 

The CDP model does not always results in proper solutions and there appeals to be some confusion among 
researchers as to what constitutes a proper solution. The CDP model is extremely parsimonious, but this is only at the 
expense of very restrictive assumptions. Even when the CDP model provides an apparently good fit to a manifest 
MTMM matrix, much of the richness of detail in the MTMM data is lost. This parsimony undermines the formative 
evaluation of MTMM data that was the reason why Campbell and Fiske (1959) first ^posed the MTMM design. 
Whereas this inevitable compromise between parsimony and detail is both a strength and a weakness of the CDP 
approach, researchers need to be aware of this consideration when interpreting the results of the CDP model. Also, tests 
of these restrictions rely heavily on the evaluation of the overall fit of the model. Thus, we find it worrisome that we 
were able to find no published applications of the CDP model that resulted in particularly poor fits. This concern was 
reinforced by the finding that the CDP model provided an excellent fit to simulated data generated from a population 
model in which assumptions underlying the CDP model were seriously violated. Whereas the fit of the CDP model was 
excellent, the CDP parameter estimates led to seriously distorted interpretations of convergent validity, discriminant 
validity, and method effects. These results may call into question the falsifiability of the model and suggest that goodness 
of fit may be a necessary but not sufficient basis of support for the CDP model. This problem was easily identified for 
the artificial data in which the underlying population model was known, but is more difficult to identify in analyses of 
real data in which researchers only have manifest MTMM matrices as a basis for evaluating CDP estimates. 

The multiple indicator approach to the CDP model apparently provides a much stronger basis for evaluating 
assumptions underlying the CDP model than the single indicator approach that is typically used. The simple first-order 
model that is the starting point in the multiple indicator approach provides a critical basis of comparison for the MI-CDP 
model. Because the MI-CDP model is nested under the corresponding first-order model, the intent of the MI-CDP is 
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merely to provide a much more parsimonious representation of the first-order model that tests the hypothesized direct 
product structure. Hence, the MI-CDP model can be rejected if its fit is substantially poorer than the first-order model or 
parameter estimates differ substantially from those in the first-order model. This provides a potential solution to two 
potential problems identified in the single-indicator CDP model. First, single-indicator CDP seems to always provide at 
least a reasonably good fit to the data - even when the CDP model is inappropriate, whereas the falsifiability of the MI- 
CDP model is easier to evaluate. Second, parameter estimates in the single-indicator model must be evaluated in relation 
to a manifest MTMM matrix, whereas the latent MTMM matrix in the multiple indicator approach provides a much 
stronger basis of comparison. 

We also have some broader, philosophical concerns about the CDP model (also see Kumar & Dillon, 1992). 
The model, at least as applied to MTMM data, is apparently based on an uncritical acceptance of the original Campbell- 
Fiske guidelines. Thus, for example, Browne (1989) noted that "Campbell & Fiske (1959) listed four requirements for 
multitrait-multimethod correlation matrices that have become generally accepted. We shall be concerned with the 
investigation of these requirements" (p. 507). Whereas we agree that the heuristic value and intent of the Campbell-Fiske 
guidelines is widely endorsed, we do not concur that their literal translation as "requirements" as embodied in the CDP 
model is widely accepted. Indeed, it is the many problems and potential ambiguities in the guidelines that has spawned 
so many alternative approaches. Whereas the application of the CDP approach certainly provides an objectivity to 
evaluating the Campbell-Fiske guidelines, it is not clear that the CDP model eliminates widely recognizxd ambiguities in 
the interpretation of the Campbell-Fiske guidelines. Furthermore, it is difficult to conceive of how multiplicative 
correlations could arise other than through a multiplicative relation of the scores which, if taken literally, would 
undermine the logic of the Campbell-Fiske guidelines and even the logic of the classical approach to test theory. 
Whereas Browne (1984) has not claimed that support for the CDP model necessarily leads to such dire consequences, we 
nevertheless find paradoxical the assumption that support for the CDP implies multiplicative relations and provides a 
basis for evaluating the Campbell-Fiske guidelines that appear to be based on an assumption of additivity that is 
invalidated by these multiplicative relation. More generally, we are loath to relinquish the many conceptual and 
theoretical advantages in the additive assumption of variance components explicit in classical test theory and 
conventional factor analysis that would have to be abandoned if such a multiplicative model were taken literally (for 
further discussion see Campbell & O'Connell, 1967, 1982; Kumar & Dillon, 1992). 

An unresolved problem is why the single-indicator CDP model is apparently able to fit data so well even when 
assumptions underlying the model are violated. Whereas we have no definitive answer to this dilemma, we suspect that 
the problem has to do with the confounding between measurement issues (communality estimates and scaling constraints 
in the E and D diagonal matrices) and the support for the direct product structure of relations among trait and method 
effects. This problem appears to be similar to that described by Marsh (1993) for single-indicator and multiple-indicator 
simplex models of longitudinal data. Rogossa and Willet (1985) claimed that satisfying the quasi-simplex structure was 
simpler than it should be, calling into question the falsifiability of the model. They demonstrated that the typical quasi- 
simplex provided a very good fit to simulated data generated from a population model that violated assumptions 
underlying the simplex model. Based on much of the same logic used to support the MI-CDP model here, Marsh (1993) 
argued for the superiority of a multiple indicator approach to simplex models. He demonstrated that the single-indicator 
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simplex model produced apparently biased estimates that exaggerated support for a simplex structure, but that the 
problem was apparently resolved by the application of the multiple-indicator simplex model. In the single-indicator 
simplex model, as in the single-indicator CDP model, structural parameter estimates are confounded with measurement 
issues. In the multiple-indicator approach, however, the measurement and structural issues are more clearly separated so 
that it was easier to reject the simplex model when support for a simplex structure is weak. 

Here we have advocated the MI-CDP model. It is, however, important to emphasize that we further recommend 
the multiple indicator approach as a basis for all MTMM studies. The usefulness of the multiple-indicator approach has 
been demonstrated with the CFA approach to MTMM data and can be generalized to any approach that begins with a 
MTMM matrix. Many of the advantages for beginning with a latent MTMM matrix instead of a manifest MTMM 
matrix in the CDP approach generalize to other approaches as well (e.g., Marsh & Hocevar, 1988; Marsh, in press). The 
major limitation of this approach is the requirement that researchers collect multiple indicators of each measured 
variable. Whereas this may not be possible in the continued reanaiyses of existing MTMM matrices that seem to 
pervade the MTMM literature, multiple indicators of each trait/method combination are typically collected in original 
studies. Indeed, undue attention should not, perhaps, be given to studies that make inferences on the basis of single-item 
scales « as opposed to those that use a single score to summarize appropriately constructed multi-item scales. 
Particularly for MTMM studies that focus on the formative evaluation and improvement of measurement instruments as 
originally advocated by Campbell and Fiske (1959), the requirement of multiple indicators should not be a problem. 

In conclusion, this investigation has an important message for applied researchers who wish to use the MTMM 
paradigm. MTMM data has an inherently complicated structure that will not be fully described in all cases by any of the 
models or approaches typically considered. There is, apparently, no "right" way to analyze MTMM data that works in all 
situations. Instead, we recommend that researchers consider several alternative approaches to evaluating MTMM data -- 
an initial inspection of the MTMM matrix using the Campbcll-Fiske guidelines followed by fitting at least the subset of 
CFA models recommended by Marsh and Grayson (in press) and the CDP model. Particularly, when researchers have 
access to raw data and there are multiple indicators of each scale/method combination, the multiple indicator approach 
described here should be used in conjunction with the Campbell-Fiske guidelines, the CFA models, and the CDP model. 
The Campbell-Fiske guidelines should be used primarily for formative purposes, the CDP model seems most appropriate 
for summative evaluations, and the CFA models apparently serve both summative and formative purposes. It is, 
however, important that researchers understand the strengths and weaknesses of the different approaches. For each of the 
different latent-variable approaches, researchers should evaluate results in relation to technical considerations such as 
convergence to proper solutions and goodness of fit, but should also place more emphasis on substantive interpretations 
and theoretical considerations. Despite the inherent complexity of MTMM data, we feel confident that the combination 
of common sense, a stronger theoretical emphasis on the design of MTMM studies, a better quality of measurement at 
the level of trait-method units, an appropriate arsenal of analytical tools such as recommended here, and a growing 
understanding of these analytic tools will allow researchers to use the MTMM paradigm effectively. Finally, researchers 
should be aware of the dangers inherent in using the CDP model to "buy" a proper solution by "selling" the ability to 
make formative evaluations of MTMM data. This problem may be compounded by the ease which the CDP model 
appears to be able to fit single-indicator data the violates assuptions underlying the model. 
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Footnotes 

1 Kumar and Dillon (1992) noted that whereas the CDP structure is strictly multiplicative, CFA models that posit trait 
and method factors can accommodate both additive and apparently multiplicative effects. That is, (he effect of a given 
method can vary for different measured variables associated with the same method. Whereas the CFA structure may be 
more flexible than the CDP structure, this greater flexibility is typically at the expense of a less of parsimony and, 
perhaps, problems associated with improper solutions (see Marsh, 1989; Marsh & Bailey, 199 1). 
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Table 1 

The Pm. Pt. and PC Matrices for a CDP Model with 3 Traits and 3 Methods 



Pm « 

1 

M21 1 

M31 M32 1 

Pt « 

1 

T21 1 

T31 T32 1 

Pc = Pin x Pt = 

IxPt 

M21xPt 1 



M31xPt 


M32xPt 


IxPt 












1 

T21 


1 














T31 


T32 


1 












M21 


M21XT21 


M21XT31 


1 










M21XT21 


M21 


M21xT32 


T21 


1 








M21XT31 


M21XT32 


M21 


T21 


T32 


1 






M31 


M31XT21 


M31XT31 


M32 


M32xT21 


M32xT31 


1 




M31xT21 


M31 


M31xT32 


M32xT21 


M32 


M32XT32 


T21 


1 


M31XT31 


M31xT32 


M3 ]. 


M32xT31 


M32XT32 


M32 


T21 


T32 



Note, Pm is the correlation matrix among method factors with a typical element being r (Mr, Ms), Pt is the correlation 
matrix of relations among latent trait factors with a typical element being r (Ti, Tj) = Tij, and Pc is the MTMM matrix 
appropriately corrected for unreliability that is defined to have the direct (kronecker) product structure Pm x Pt, Thus, for 
example, T21 is the correlation between traits 1 and 2, M12 is the correlation between methods 1 and 2, and T21xM31 is 
the product of the correlation between traits 1 and 2 and the correlation between methods 3 and 1 . Hence, the 36 off- 
diagonal elements in Pc are defined in terms of only 6 estimated parameters (M21, M31, M32, T21, T31, and T32); all 
elements are either equal to one of these six parameters or are the product of two of these six parameters,. 
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Table 2 

The MTMM Correlation Matrix for the Bvrne Data 
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tlml 


(.89) 
















t2ml 


.384 


(.79) 














t3ml 


.441 


.002 


(.92) 












tlm2 


.622 


.368 


.353 


(.84) 










C2m2 


.438 


.703 


.008 


.441 


(.89) 








t3m2 


.465 


.069 


.871 


.424 


.136 


(.95) 






Clm3 


.678 


.331 


.478 


.550 


.380 


.513 


(.87) 




r.2m3 


.458 


.541 


.057 


.381 


. 658 


.096 


.584 


(.90) 


KM 


.414 


,027 


.825 


,37,? 


,02? 


. 810 


.592 


, 3 35 



Note. tl=gcncral school self-concept, t2=verbal self-concept, t3=math self-concept; ml, m2, and m3 are three different 
self-report instruments This MTMM matrix is derived from data presented by Byrne and Shavelson (1986). Values in 
parentheses, coefficient alpha estimates of reliability, were not actually used in the analysis. 
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Table 3 

Parameter Estimates For the CDP Model and the CDP Model Wilh a Kronccker Error Structure 



CDP Model 

Trait Correlations (Pt) 
Tl 1 

T2 .636 1 
T3 .592 .151 1 
Communal ity Indices 
T1M1 T2M1 T3M1 T1M2 T2i 
.766 .901 .963 .879 .8 



Method Correlations (Pm) 
Ml 1 

M2 .959 1 

M3 .834 .837 1 

! T3M3 T1M3 T2M2 T3M3 a 
' .953 .939 .900 1.00 



9 

ERLC 



CDP Model With a Kronecker Error Structure 

Trait Correlations (Pt) Method Correlations (Pm) 

Tl 1 Ml 1 

T2 .694 1 M2 .960 1 

T3 .596 .138 1 M3 .848 .831 1 

Communal ity Indices 

T1M1 T2M1 T3M1 T1M2 T2M2 T3M3 T1M3 T2M2 T3M3 

.810 .827 .953 .842 .857 .962 .927 .935 .985 
a This error variance corresponding to this communality was constrained to be nonnegative. Because the E matrix in 
the the CDP model was not positive definite, the solution is improper. 
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Table 4 

Simulated mtmm matrix used to Evaluate the cdp Model 



ERLC 



MTMM Motrin 










f 1 ml 

l inn 


1 nnn 

1 • u u u 










c <&mi 




1 , 000 








f 'Jrn1 

l j mi 


• U j / 


.054 


1.000 






tlni2 


.720 


. 384 


.024 


1.000 




t2m2 


.432 


.640 


.024 


.424 


1.000 


t3m2 


.027 


.024 


.090 


.084 


.084 1.000 


tlm3 


.630 


.336 


.021 


.560 


.336 .021 1.000 


t2m3 


.378 


.560 


.021 


.336 


.560 .021 .314 1.000 


t3rn3 


.045 


.040 


.150 


.040 


.040 .150 .065 .095 1.000 



CDP Parameter Estimates 

Trait Correlations (Pt) Method Correlations (Pm) 

Tl 1.000 Ml 1.000 

T2 .597 1.000 M2 .934 1.000 

T3 .160 .179 1.000 M3 .943 .904 1.000 
Communality Indices 

T1M1 T2M1 T3M1 T1M2 T2M2 T3M3 T1M3 T2M2 T3M3 

.912 .810 .313 .845 .845 .342 .732 .733 .496 

Note: The MTMM matrix was simulated from a confirmatory factor model with 3 correlated trait factors, 3 uncorrected 
method factors, and 9 uncorrected uniquenesses. For each measured variable the trait-factor loading, method-factor 
loading, and uniqueness was: .9, .1, .18 (T1M1); .8, .1, .35 (T2M1); .3, .3, .82 (T3M1); .8, .2, .32 (T1M2); .8, .2, .32 
(T2M2); .3, .3, .82 (T3M2); .7, .1, .50 (T1M3); .7, .2, .47 (T2M3); .5, .3, .66 (T3M3). The three trait correlations were .6 
(T1T2), .1 (T1T3), and .1 (T2T3). The MTMM matrix that was then evaluated with the CDP model. Even though the 
true trait variances for Tl and T2 differed substantially from T3, apparently violating an assumption of the CDP model, 
the CDP model fit the data very well (X 2 (21) = 1 1.15 for a hypothetical N=500). 
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Tables 

Completely Standardized Solutions For the First-Order Model and Multiple-Indicator CDP Model 

First -Order Model Multiple Indicator CDP Model 

Measured Factor Loadings Factor Loadings 



Variables 


T1M1 


T2M1 


T3M1 


T1H2 


T2M2 


T3M2 


T1M3 


T2M3 


T3M3 


Uniq 


T1M1 


T2M1 


T3M1 


T1M2 


T2M2 


T3M2 


T1M3 


T2M3 


T3M3 


Uniq 


tlml-1 


.83 


.00 


.00 


.00 


.00 


.00 


.00 


.00 


.00 


.32 


.84 


.00 


.00 


.00 


.00 


.00 


,00 


.00 


.00 


.30 


tlml-2 


,P8 


.00 


.00 


.00 


.00 


.00 


.00 


.00 


.00 


.22 


.88 


.00 


.00 


,00 


.00 


.00 


,00 


.00 


.00 


.22 


tlml-3 


,88 


.00 


.00 


,00 


.00 


.00 


.00 


.00 


.00 


.23 


.87 


.00 


.00 


.00 


.00 


,00 


.00 


.00 


.00 


.24 


t2ml-l 


,00 


.81 


.00 


.00 


,00 


.00 


.00 


.00 


.00 


.35 


.00 


.80 


.00 


.00 


.00 


.00 


.00 


.00 


.00 


.37 


c2ml-2 


.00 


.73 


.00 


.00 


.00 


.00 


,00 


.00 


.00 


.47 


.00 


.73 


.00 


.00 


.00 


.00 


.00 


.00 


.00 


.46 


t2ml-3 


,00 


,66 


.00 


.00 


.00 


.00 


.00 


.00 


.00 


.56 


.00 


.66 


.00 


.00 


.00 


.00 


.00 


.00 


.00 


.57 


t3ml-l 


.00 


.00 


.93 


.00 


, uu 


.00 


,0u 


.ou 


. uu 


. 13 


,00 


.00 


.92 


.00 


. uu 


.00 


.00 


.00 


.00 


.16 


i-3ml-2 


,00 


.00 


.92 


.00 


.00 


.00 


.00 


, 00 


.00 


. JG 


.00 


,00 


.90 


,00 


.00 


.00 


.00 


.00 


.00 


. 19 


r.3ml-3 


.00 


.00 


.87 


.00 


.00 


.00 


.00 


.00 


.00 


.24 


,00 


.00 


.85 


,00 


.00 


.00 


,00 


,00 


.00 


.28 


tllft2-l 


,00 


.00 


.00 


.89 


,00 


.00 


.00 


.00 ' 


.00 


.20 


,00 


.00 


.00 


,87 


.00 


,00 


.00 


.00 


.00 


.24 


c.lm2-2 


.00 


.00 


.00 


.73 


.00 


.00 


.00 


,00 


.00 


.46 


.00 


.00 


.00 


.73 


.00 


.00 


. 00 


.00 


.00 


.47 


t lm2-3 


.00 


.00 


.00 


.80 


.00 


.00 


.00 


.00 


,00 


.36 


,00 


. 00 


.00 


.79 


.00 


,00 


.00 


.00 


.00 


.37 


t2m2-l 


.CO 


.00 


.00 


.00 


.54 


.00 


.00 


.00 


,00 


,71 


.00 


.00 


.00 


.00 


. 55 


.00 


.00 


.00 


.00 


.69 


t2ro2-2 


.00 


.00 


.00 


.00 


.85 


.00 


.00 


,00 


.00 


,27 


,00 


.00 


.00 


. 00 


.86 


.00 


.00 


.00 


.00 


.26 


t2m2-3 


.00 


.00 


.00 


.00 


.91 


.00 


.00 


,00 


,00 


.18 


,00 


,00 


,00 


,00 


.91 


.00 


,00 


.00 


.00 


.10 


t3rn2-l 


.00 


.00 


.00 


.00 


.00 


.93 


.00 


.00 


,00 


.14 


,00 


,00 


.00 


,00 


.00 


.92 


.00 


,00 


,00 


.16 


t3rn2~2 


,00 


.00 


.00 


.00 


.00 


,90 


.00 


.00 


.00 


.19 


.00 


,00 


.00 


,00 


.00 


.88 


,oc 


.00 


.00 


.23 


t3m2-3 


,00 


.00 


.00 


.00 


.00 


.88 


.00 


.00 


.00 


.22 


.00 


. 00 


.00 


. 00 


.00 


.87 


.00 


.00 


.00 


.25 


c lm3 - 1 


, 00 


.00 


.00 


.00 


.00 


.00 


.80 


,00 


,00 


,36 


,00 


,00 


.00 


,00 


.00 


.00 


.77 


.00 


.00 


.40 


tlm3-2 


.00 


.00 


.00 


.00 


.00 


,00 


.90 


.00 


.00 


,20 


,00 


.00 


.00 


.00 


,00 


,00 


.90 


.00 


.00 


. 19 


tlm3-3 


,00 


. 00 


,00 


.00 


.00 


.00 


.78 


.00 


.00 


,39 


.00 


.00 


.00 


,00 


,00 


.00 


,78 


.00 


,00 


,39 


t2m3-l 


.00 


.00 


.00 


.00 


.00 


.00 


,00 


.80 


.00 


.36 


.00 


,00 


.00 


,00 


.00 


.00 


.00 


,81 


.00 


.35 


t2m3-2 


.00 


.00 


.00 


.00 


.00 


.00 


,00 


.93 


.00 


. 14 


.00 


.00 


.00 


.00 


.00 


.00 


.00 


.93 


.00 


. 13 


t2m3-3 


,00 


.00 


,00 


.00 


.00 


.00 


,00 


.83 


.00 


.32 


.00 


.00 


.00 


.00 


. 00 


.00 


.00 


,83 


,00 


.31 


t3m3-l 


.00 


.00 


.00 


,00 


. 00 


.00 


.00 


.00 


.95 


.10 


,00 


.00 


.00 


.00 


.00 


.00 


.00 


.00 


.94 


.12 


t3m3-2 


,00 


,00 


.00 


,00 


.00 


.00 


,00 


.00 


,96 


.08 


.00 


.00 


.00 


.00 


,00 


,00 


.00 


.00 


.95 


. 10 


t3m3-3 


.00 


.00 


.00 


.00 


. ou 


,00 


.00 


.00 


.87 


.25 


,00 


.00 


,00 


,00 


,00 


,00 


.00 


,00 


.83 


.31 


Factor correlations 






































T1M1 










































T1M2 


,46 




















.37 





















T1M3 


.48 


-.02 


















,37 


.23 


















T2M1 


.77 


.33 


.39 
















86 


,32 


.32 
















T2M2 


.43 


,81 


- .01 


.40 














jj 


.86 


,20 


, 37 














T2M3 


.49 


. 03 


.95 


.46 


.08 












.32 


.20 


,86 


.37 


.23 












T3M1 


,78 


.39 


.53 


,71 


.39 


.55 










.77 


.29 


.28 


,78 


.29 


.29 










T3M2 


,52 


,65 


.05 


.43 


.75 


.07 


.64 - 








.29 


.77 


,18 


.29 


.78 


. 18 


.37 








T3M3 


.46 


.02 


.87 


.43 


.01 


.87 


.65 


. 14 - 






.28 


.18 


.77 


.29 


,1« 


.78 


,2.7 


.23 







Note. The goodness of fit indices are better for the first-order model than for the multiple-indicator CDP model: X z 
(288) = 1506 vs. X 2 (318) = 2100; RNI = .939 vs. .911. The factor correlation matrix for the CDP model is the direct 
product of a Pm matrix of correlations among the methods [convergent validities; having off-diagonal elements of .86, 
.77 and .78] and a Pt matrix of correlations among traits [having off-diagonal values of .37, .37, and .23]. 
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Note. The model was fit to a 27x27 covariance matrix. Plausible starting values were also provided that are not 
shown to conserve space. 

1 In PA PS (the pattern matrix psi of factor correlations), all pattern elements greater 1 arc used to impose 
equality constraints (i.e., all parameter estimates with pattern elements of 2 are constrained to be equal). 2 
Additional constraints are imposed to fit the CDP model by requiring some fixed parameters to be equal to the 
product of two parameter estimates (see Table 1). 
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